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Abstract 

We consider ergodic backward stochastic differential equations in a dis¬ 
crete time setting, where noise is generated by a finite state Markov chain. 

We show existence and uniqueness of solutions, along with a comparison 
theorem. To obtain this result, we use a Nummelin splitting argument 
to obtain ergodicity estimates for a discrete time Markov chain which 
hold uniformly under suitable perturbations of its transition matrix. We 
conclude with an application of this theory to a treatment of an ergodic 
control problem. 
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1 Introducing Discrete Time BSDEs 

1.1 Introduction 

Over the last 25 years, Backward Stochastic Differential Equations (BSDEs) 
have been extensively researched, and established as a fundamental object in 
mathematical finance and stochastic control. Here, one is typically interested 
in the solution process (Y, Z) of an equation of the form 

U - ^ /(w, u, Yu, Zu)du + ZudWu = C, 

for some generator (or driver) function / and some JA-measurable terminal 
condition U where W is an n-dimensional Brownian motion. 

When the cost of a control problem is considered in an ergodic manner, that 
is, it is the long run behaviour which is important, the associated equations turn 
out to be examples of a variation of these equations, named ‘Ergodic BSDEs’, 
which typically take the form 

YT=Yt- [ (/(W, Zu) - X)du + [ ZudWu. 

Jt Jt 

* Research supported by the Oxford-Man Institute for Quantitative Finance. 


1 



Having been first introduced in Fuhrman, Hu and Tessitore cni, the theory of 
these equations and their link to ergodic control problems have been researched 
over the last few years; see Richou [13], Debussche, Hu and Tessitore [5] and 
Cohen and Fedyashov [Sj. In Cohen and Hu [B], these equations were consid¬ 
ered in the case where noise is generated by a continuous time, countable state 
Markov chain. 

This paper considers the discrete time analogue of these equations, where 
noise is generated by a discrete time, finite state Markov chain. In Section 2 
we shall extend the existing theory of discrete time BSDEs, as given in Cohen 
and Elliott |3], to a class of infinite horizon ‘discounted’ BSDEs. Section 3 is 
a brief digression to the study of uniformly ergodic Markov chains in discrete 
time, in which we shall show how to obtain ergodicity estimates for a Markov 
chain which are robust under a suitable perturbation of its transition matrix. 
These estimates have not been previously obtained in a discrete time setting 
and are of independent interest in the study of the ergodic properties of discrete 
time Markov chains. 

We shall then make use of these estimates in Section 4 to construct a suitable 
limiting procedure with which to prove the existence of bounded Markovian 
solutions to discrete time Ergodic BSDEs. We will also prove a comparison 
theorem, and observe the relationship between the Markovian solution and the 
ergodic measure of the underlying Markov chain. Finally, in Section 5 we shall 
show how our theory can be applied to a particular ergodic control problem. 

1.2 Finite horizon BSDEs 

We will begin by introducing the theory of finite horizon, discrete time BSDEs, 
as established in [3]. 

Consider an underlying discrete time, finite state stochastic process X. 
Without loss of generality, we may assume that X takes values in the stan¬ 
dard basis of for some N gN. That is, for each t G {0,1,2,...}, 

Xt G X := {ei, 62 ,..., e^v}, 

where = (0, 0,..., 0,1, 0,..., 0)* G and [ • ]* denotes vector ( or matrix) 
transposition. 

Let (D, {J^t}t>o,]P) be a filtered probability space, where Ft is the com¬ 

pletion of the tr-algebra generated by the process X up to time t, and assume 
that F = Foe ■■= Vt>o 

Remark l.I. We have not assumed that F is complete, nor that Fq contains all 
the P-null sets in Foo- Later, we will consider our processes under a variety of 
different probability measures, and we do not wish for P-null events in the tail 
CT-algebra to be known at finite times, as these events may occur with positive 
probability under a new measure. 

Note that the atoms of Ft are sets of positive measure of the form 

{uj' G D : Xs(uj') = Xs(uj) for all s < t} 

for some w G D, up to a null set in Ft- We will find that much of our analysis 
can be done by restricting our attention to individual atoms. 

For t > 1, let Mt := Xt — ]E[Xi | which we note defines a martingale 

difference sequence M. 
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We now define a backward stochastic difference equation (BSDE) as an equa¬ 
tion of the form 

Yt- Z:Mu+i=^, (1.1) 

t<U<T t<U<T 

where T > 0 is a finite deterministic terminal time, / : fl x {0,..., T — 1} x 
R X ^ R is an adapted map, and ^ : fl —>■ R is an .FT-measurable random 
variable. By “/ is adapted” we mean that, for all t G {0,..., T — 1}, the map 
from n X R X R^ ^ R given by (w, y, z) ^ /(w, t, y, z) is Ft ® yB(R) ® ,B(R^)- 
measurable. Since there are only finitely many possible paths for X up to time 
t, Ft is sufficiently coarse that all J't-measurable real-valued functions belong 
to J't,P) and L°“(n, J't.P). 

A solution to the BSDE (lEB is a pair (F, Z) of adapted processes, taking 
values in R X R^, such that 

Yt{u:)- Y, f{u:,u,Y^{u:),Z^{u:))+ Y Zl{u:)Mu+i{u:) = i{uj) 

t<u<T t<u<T 

holds for alH G {0,..., T} and almost all oj. 

Definition 1.2. We will denote by || • ||Mt+i the stochastic seminorm on the 
space of R^-valued, J^t-measurable random variables given by 

:= Z;E[Mt+,M,\, I Ft] Zt = E[iZ:Mt+ir \ F ], 

and write Zt ~Mt+i Z'^ whenever \\Zt — Z^\\Mt+i = 0 a.s. We also denote by 
II • ||m the seminorm on the space of adapted processes in R'^ given by 

E = E mz:M^+,f], 

0<u<T _ 0<u<T 

and write Z Z' whenever ||F — F'||m = 0. 

It is clear that Z Z' if and only if Zt ^Mt+i Z[ for alH G {0,..., T — 1}. 
It is also easy to see that ~m and ~Mt+i are both equivalence relations. 

We will construct the solutions of our BSDE by making use of the following 
Martingale Representation Theorem from Elliott and Yang [S]. 

Theorem 1.3. Let L be a real-valued {Ft}t>o-adapted martingale. Then there 
exists an adapted -valued process Z such that 

Lt = Lq + E! Z*Mu+i 

0<u<.t 

for all t. Further, the process Z is unique up to equivalence ~m. 

Proof. By the Doob-Dynkin Lemma (see for example Shiryaev [m p.174]), for 
each t, there exists a function gt:XxXx---xX^R such that 

Lt+i = Lt -\- gt{Fo, Xi,..., Xt+i). 

Define Zt as the -measurable, R^-valued random variable with components 
given by e*f.Zt = 5 t(Yo, Xi,... ,Xt,ek) for each G A, and notice that 

ZtXt+i = gt(Xo,Xi,... ,Xt, Xt+i) = Lt+i — Lt. 


II^IIm:=E 
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As L is a martingale, we see that Z^Mt+i = Z^Xt+i- Hence, Lt+i = Lt + 
Z^Mt+i, and the result follows. The uniqueness of Z follows easily from the 
definition of ^m- d 

Corollary 1.4. Let W be a real-valued tFt+i-fneasurable random variable such 
that = 0. Then there exists an -valued Xt-measurable random 

variable Zt such that 

W = Z;Mt+i a.s. 

Further, this variable is unique up to equivalence ~Mt+i ■ 

Proof. Simply consider the martingale L defined by Lg = l{s>t}lT, and apply 
Theorem 11.31 □ 

The following theorem provides sufficient conditions for the existence of a 
unique solution to our finite horizon BSDE. In general, these conditions are also 
necessary for a unique solution to exist for all terminal conditions as shown 
in [3J Corollary 2]. 

Theorem 1.5. Suppose our driver f is such that the following two assumptions 
hold: 

(i) For any Y, if Z Z', then f{oj,t,Yt,Zt) = f{uj,t,Yt,Z^) a.s. for all t. 

(ii) For any z £ , any t and for almost all oj, the map 

y^y- fi^,t,y,z) 

is a bijection R —>■ K. 

Then, for any J-t- measurable terminal condition the BSDE CJP has an 
adapted solution (Y,Z). Moreover, this solution is unique, up to indistinguisha- 
bility for Y and for Z. 

Proof. We shall construct the solution using backward induction. Note that 
Yt = f is the unique solution for Y at time T. Suppose that we have found a 
solution for Y at time t + 1. The one step dynamics of HH) are given by 

Yt - f{uj,t,YuZt) + ZfMt+i = Yt+i. ( 1 . 2 ) 

Taking an E[ • | Ft] expectation gives 

Yt - fitu, t, Yt, Zt) = E[rt+i I Ft], (1.3) 

and substituting this back into (HH), we obtain 

ZfMt+i = Yt+i - E[yt+i I Ft]. (1.4) 

It follows that any adapted solution {Yt,Zt) at time t must satisfy both (11.31) 
and mi). By Corollary 11.41 there exists an M^-valued J^-nieasurable random 
variable Zt such that dm holds, and this variable is unique up to equivalence 

With this Zt, it follows from assumption (z) that equation dm is now 
uniquely determined as an equation in Yt. By assumption (ii), for almost all w, 
this equation has a unique solution Yt{ui). The pair (Yt,Zt) is then the unique 
solution of the BSDE at time t, and we can therefore construct the full solution 
(Y, Z) by backward induction. Since for each t, Yt is unique up to equality a.s. 
and Zt is unique up to it follows that the solution {Y,Z) is unique up 

to indistinguishability for Y, and for Z. □ 
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1.3 BSDEs on Markov chains 


Let us now consider the case when our underlying process X is a discrete time, 
finite state Markov chain under P. To keep our notation somewhat consistent 
with the continuous time theory in [^, we shall define the transition matrix 
At = [aij] of the Markov chain X by = P(Xt+i = i \ Xt = j), so that 
the columns of the matrix give the probability distributions associated with the 
transitions of the chain. Note that X is allowed to be time-inhomogeneous, so 
that its transition matrix At can vary (deterministically) through time. 

Notice that now E[Xt+i | Tt] = AtXt, so that Mt+i = Xt+i — AtXt. 

Definition 1.6. We shall say that a driver / is Markovian if at every time 
t, the value of / only depends on w through the value of Xt- That is, / is 
Markovian if it can be written as f{u},t,y,z) = f{AitA,yTz) for some function 
/:T’xNxRxK^-i>IR. 

Lemma 1.7. Assume that X is a Markov chain with transition matrix At, 
and let f be a Markovian driver which satisfies the conditions of Theorem \1.5\ 
Suppose that f = lp{Xt) for some deterministic function : A:’ —>■ R, and let 
{Y, Z) be the solution of the corresponding BSDE Then there exists a 

function v : {0,1,..., T} x T” —>■ K such that, for each t, Yt = v(t, Xt) and the 
k*^ component of Zt is given by e’^Zt = v{t -\- 1,6^) (cf. Theorem 3.2]). 

In particular, the solution (Y, Z) is also Markovian in the sense that Yt and 
Zt are both deterministic functions of Xt (and in fact Zt is completely deter¬ 
ministic). Moreover, writing Vt+i for the vector in with entries e^Vt+i = 
v{t 1, e/c), the function v satisfies 

v{t, Ck) - f{ek, t, v{t, Cfc), vj+i) - Vj+i AtCfc = 0 (1.5) 

for all t G {0,..., T — 1} and all ek G X such that P(Xt = Ck) > 0. 

Proof. We shall prove the existence of the function v by backward induction in 
t. Let v{T, •) :=(/?(•), so that Yt = v{T, Xt). 

Suppose that for some t we have found a function u(t -I- 1, •) : T” —>• R such 
that Yt+i = v{t l,Xt+i). Recall from the proofs of Theorem 11.31 and Theo¬ 
rem [T^ that the kf^ component of Zt is given by e),Zt = gt{Xo, Xi,... ,Xt,ek), 
where, in this case, 

N 

gt{Xo,Xi,..., Xt, Xt+i) = v{t + 1, Xt+i) -J2v{t + 1, ei)e*AtXt. (1.6) 

Note that the sum on the right-hand side of (HSl) does not depend on Xt+i, 
so that this term corresponds to the addition of a multiple of 1 to Zt, where 
1 denotes the vector in R^ with all entries equal to 1. However, we infer from 
Definition o that ||l||Mt+i = 0, so the addition of multiples of 1 does not 
change the value of Zt up to equivalence We may therefore ignore this 

term, and deduce that = v(t -h 1, e/c) for all Ck G X. 

By assumption, the driver / can be written as f(uj,t,y,z) = f{Xt,t,y,z) 
for some function /. From the proof of Theorem 11.51 Yt is given by the (a.s. 
unique) solution of the equation: 

Yt - f{Xt,t, Yt, Zt) = E[y*+i I Ft]. 
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Hence, since Zt and E[y(_|_i | iFt\ are both deterministic functions of Xt, it follows 
that the same is true of Yt, so there must exist a function v{t, •) : A” —)• R such 
that Yt = v{t,Xt). We therefore deduce the existence of the required function v 
by backward induction. The equation follows from the one-step dynamics 

of dni). □ 

2 Infinite Horizon BSDEs 

2.1 Discounted BSDEs 

We now introduce a ‘discounted’ BSDE as an infinite horizon equation of the 
form: 

YT = Yt- ^ {fico,u,Z^)-aY^)+ ^ ( 2 . 1 ) 

t<u<T t<u<T 

where a > 0 is a fixed constant. Here T is no longer a fixed terminal time. 
Instead, both t and T may take any finite values such that 0 < t < T. A 
solution of this equation is an adapted process {Y, Z) such that (12.ip holds a.s. 
for every 0 < t < T. In this section we will show that this equation admits 
unique bounded solutions for a suitable class of drivers. Similar results are 
considered in a continuous time setting in Royer |14] and in [Q, however the 
discrete time case has not been previously considered. 

We will return to the case when X is a Markov chain in Section 2.2. However, 
the theory we present here for discounted BSDEs holds much more generally. 

Remark 2.1. Recall Definition o In light of our infinite horizon setting, we 
will henceforth write Z Z' if Zt Z[ for all t > 0. 

Proposition 2.2. Suppose that for any t > 0, y G M., z, z' € R'^ and any atom 
F G Xt, the driver f satisfies 

> min {{z - z')*{e^ -E[X(+i |Xt])} (2.2) 

iGjf 

on F, where jf := {i : P(Xt_|_i = | F) > 0}, unless 

min{(z-z')*(e.-E[Xi+i|Ft])} =0, 
ieJf 

in which case the inequality can be replaced by an equality. Then, for any adapted 
processes Y, Z, Z', there exists a probability measure Q on (O, F), which is equiv¬ 
alent to P on Ft for every finite t, and such that 

Mf.= - Y. - f{u:,u,Y,,,Z',f)) Y Y [Zu - Z'J* 

0<u<t 

is a martingale under Q. 

Proof. We will define Q inductively in t. Eirst let Q|.Fo = P|.Fo- 

Next, suppose that we have defined Q on Ft for some t > 0. Let F be an 
atom of Ft (so that Q(F) > 0), and let 

Ff := {{Zt - Zfne, - Ep[W+i | Ft]) : ^ € jf}, 
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where here all variables are evaluated on F. Note that 


Ep[(Zt - Z[)*{Xt+i - Ep[Xt+i I Ft]) I = 0, (2.3) 

and hence that min (S'f’) < 0 and max > 0. Suppose that min {sn < 0. 
By assumption, we can swap the roles of z and z' in (I2.2I1 . and deduce that 

/(w, t, Yt, Zt) - f{uj, t, Yt, Z') e conv° (S'f), 

where here conv°(-) denotes the interior of the convex hull of a subset of K.. 

If min {S^) = 0, then it follows from (12.3|) that = {0}, and hence, by 
assumption, that 

f{uj,t,Yt,Zt)- Yu Z',) = 0. 

It follows from the definition of convexity that, in either case, for each i G jjf, 
there exists /if > 0 such that ~ 

f{u;,t,YuZt)-f{oj,t,Yt,Z'^)='£t^fiZt-Z'^)*{e,-Ep[Xt+i\Ft]). (2.4) 

iejtf 

Define Q({Xt+i = eJnF) := /if(Q(F) for i G jjf, and Q({Xt+i = eJnN) := 0 
for i ^ Jjf. 

The measure Q is now defined on the whole of Ft+i and therefore, by in¬ 
duction, on Us> 0 '^s- Caratheodory’s extension theorem (see for example 
[13 p.l52]), there exists a unique extension of Q to a probability measure on 
F = o'(lJ^>Q J^s). By construction, Q is equivalent to P on Ft, for any t > 0. 
Further, for any t, F and i G Jf as above, Q({Xt+i = e^} | F") = /if. 

Note that 

EQ[Mt+i - Mt I Xt] = -{f{u,t,Yt,Zt) - f{w,t,Yt,Z[)) 

+ {Zt - Z[r (Eq[X*+i I Ft] - Ep[Xt+i I Ft]). 

It follows from that the above expression is equal to zero on every atom 
F oi Ft- Hence, as required, M is a martingale under Q. □ 

Remark 2.3. Note that the measure Q constructed in the proof of Proposition !^^ 
is by no means uniquely defined, as in general there will be many ways of 
choosing the constants /if such that the required conditions hold. We also note 
that in general Q will not be equivalent to P on F, which is why it was important 
that we did not assume that Fo contains all the P-null sets in F. 

Definition 2.4. We shall say that a driver / is Lipschitz in z, if there exists 
L > 0 such that, for any t and any -measurable random variables Yt, Zt, Z[, 

\f{uj,t,Yt,Zt) - f{uj,t,Yt,Z{)\< L\\Zt- a.s. 

We will now proceed to prove a result on the existence and uniqueness of 
bounded solutions to our discounted BSDE. The proof is an adaptation of that 
of 13 Theorem 2.10] into our discrete time setting. 

Theorem 2.5. Let a > 0, and suppose that the driver f : LI xN x —>■ R. is 

independent of y, and satisfies the following conditions: 
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• f is Lipschitz in z, with Lipschitz constant L, 


• f satisfies the conditions of Proposition \2.21 and 

• f{ui,t,Q) is uniformly bounded by some C > 0. 

Then there exists an adapted solution {Y, Z) to the (infinite horizon) discounted 
BSDE 

YT = Yt- Y. {f{io,u,Z^)-aYu)+ Y ^uMu+i, ( 2 . 5 ) 

t<u<T t<u<C.T 

satisfying \Yt\ < C/a for all t, and this solution is unique among bounded 
adapted solutions. 

Furthermore, if (Y"^, Z"^) denotes the (unique) adapted solution to the finite 
horizon BSDE 


0 = Yt^- Y {fi^,u,Z//)-aYj)+ Y {Z^YMu+i, (2.6) 

t<u<T t<u<T 

then Y^ —^ Yt in as T ^ oo, uniformly on finite sets in t. 

Proof. First we will show that if a bounded solution exists, then it is unique. 
Suppose we have two bounded solutions (Y, Z) and {Y', Z') to ()2.5I) . Let 5Y = 
Y — Y' and bZ = Z — Z', and consider the one-step dynamics, given by 

= (1 + a)bYt - {fitu, t, Zt) - f{Lu, t, Z')) + 5Z/Mt+i. (2.7) 

By Proposition [121 there exists a probability measure Qi such that 

- Y (/(^>'“>^«) -/(^>^>^«)) + Y ^Z*Mu+i 

is a martingale under Qi. It follows that 6Yt = (1 -I- a)“^EQji5yt+i | J't], and 
hence, by induction, that 

\SYt\ < (l + a)-(’'-‘)EQ,[|5i;| I J-*] 

for all r > t. Note that the above holds Qi-a.s., but since Qi is equivalent 
to P on Et for every finite t, there is no distinction here between Qi-a.s. and 
P-a.s. Since Y and Y' are both uniformly bounded, the same is true of 6Y. We 
then deduce, upon taking the limit as r —>■ oo in the above, that SYt = 0 a.s. 
It follows that Y = Y', up to P and Qi-indistinguishability. Substituting back 
into (12.7|) . we deduce that SZ/Mt+i = 0, and hence that Z Z'■ 

We now proceed to prove existence. It follows from the Lipschitz condition 
that the driver of (12.61) satisfies the assumptions of Theorem ll.51 so the solution 
(Y’^,Z’^) exists uniquely. First we show that is bounded. 

By Proposition 12.21 there exists a probability measure such that 

- ^ {f{u;,u,Z//)-f{u;,u,0))+ Y {^1)* 

is a martingale under Qj, where we let Zj = 0 for t > T. It follows from the 
one-step dynamics of (ESI) that 

Y^ = (1 + a)-iEQT + /(w, t, 0) I Et] . 
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Using the above as the basis of an inductive argument yields 


Yt =Ec 


(1 




r — t—1 

E 


+ Ey (1 + Ck) ^ 0) 


Tt 


for all t < r < T. Setting r = T and using the fact that = 0, we deduce 
that is uniformly bounded by C/a. 

Let T' > T. Taking the difference of the one-step dynamics of and 
we obtain 


— 


y^i = (1 + a) {Y^ - Y^') - t, Zj) - f{uj, t, ZT)) (2.8) 

+ {zj-zryM,^,. 


By Proposition [221 there exists a probability measure Q 


T,T' 

3 


such that 


0<u<Ct 0<u<t 


^T,T' 


is a martingale under Qg ’ . It follows by induction that 

Yy - yy = (1 + [Yy - Yy \ , 

for all t < r < T. Then, by the boundedness established above, 


|y*^-yy| + 




T' I 


Tt 


< 




a 


Hence, we see that is a Cauchy sequence in T with respect to the L°° 

norm. Therefore, for any t > 0, there exists an J^t-measurable random variable 
y*, such that Yy —>■ Yt in L°° as T —>■ oo, and this convergence is clearly uniform 
on finite sets in t. Further, Y = {yt}t>o is also uniformly bounded by C/a. 

Taking an ]Ep[ • | Ft] expectation in (12.8L we deduce that 


{zy - zy yut+y < yy, - Yy+y + e, 


Pi+l 


-y 


T' 


t+l| 


Ft 


and it follows that [(^Zy)*Mt+i} is a Cauchy sequence in T. Hence, there 
exists an -measurable random variable Ut+i such that yyyMt+i —>■ Ut+i 
in L°° as T —>■ oo. By Corollary 11.41 there exists an J't-measurable random 
variable Zt such that Ut+i = Z/Mt+i. 

Recall the one-step dynamics of (EH), given by 

Yti, = Yy - (/(cc, t, zy) - aYy) + {zyyut+i. (2.9) 

It follows from the Lipschitz condition that f{ijj,t,Zy) f{uj,t,Zt) in as 
T —^ 00 . Finally, taking the limit as T —> oo in (1^ . we deduce that 

Yt+i =Yt- (/(w, t, Zt) - aYt) + Z/Mt+i, 

and that this holds for every t > 0. Note that these are the one-step dynamics 
of (12.Sp . Hence, (Y,Z) is a bounded adapted solution of (12.5L which we know 
is unique among such solutions. □ 


9 












2.2 7 "t>alanced drivers 

We now return to the case when our underlying process X is a (possibly time- 
inhomogeneous) Markov chain under P, with associated transition matrix At. 

Definition 2.6. Let A and B be transition matrices and let 7 S (0,1). We 
shall say that B is 7 -controlled by A (or B is controlled by A with constant 7 ), 
and write B A, A B > ')A componentwise, li B A and A>^ B then we 
will write A B. 

Remark 2.7. Writing A = [a^] and B = [ 6 ^], it is easy to check that the 
statement B A is equivalent to the condition 


7 < inf 


Qij 


i,j such that atj 



We also note that A 
entries, and 


'^.y B if and only if A and B have the same pattern of zero 


^e[7,7-'] 


for all i,j such that 7 ^ 0 . 


Definition 2.8. We shall say that a driver / is 7 -balanced if for some 7 G (0,1), 
there exists a random field ijj : ^ x N x x —>■ R^ which is adapted (in 
the same sense as /), and such that for any t > 0, y G R, z, G R'^ and for 
almost all ui, we have 


(i) f{u},t,y,z) - 


{z - z'Y t, z,z')- AtXt), 


(ii) for each a G X, 


e*Y{^,t,z,z') 

e*AtXt 


e [7,7 


where here 0/0 := 1 , and 


(iii) l*i/)(a;, t, z, z') = 1 , where we recall that 1 denotes the vector in R^ with 
all entries equal to 1 . 


Lemma 2.9. If a driver f is "f-balanced, then it is Lipschitz in z, with Lipschitz 
constant I/ 7 . 

Proof. Let ip = p{uj,t,z,z') be the random field associated with / in Defini¬ 
tion [5)^ Let t > 0 and let F be an atom of Ft. Suppose that Xt takes the 
value Cj on F. Writing At = [a^] and p = [pi], we note, from property (ii) of 
DefLnition l2.81 that pi < 'y~^aij. Then, for any -measurable random variables 
Yt,Zt, Z[ evaluated on F, we have 


\f{uj,t,Yt,Zt)- f{uj,t,Yt,Z’t)\ = 


N 


Y,PPZt-Z'tY{e.-Ate,) 


2=1 


N 


\{Zt-Z'tY{e^-AteY\ = -¥\\{Zt-Z'tYMt+i\ 


i=l 


F 


This holds on every atom F oi Ft, and the result follows. 


□ 
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Lemma 2.10. If f is a ^-balanced driver, then f satisfies the conditions of 
Proposition \2.‘d[ 

Proof. Let if = ip{uj,t,z,z') be the random field associated with / in Defini- 
tion l2.8l Let t > 0, y G M, z, z' G and let F be any atom of Ft- Recall the no¬ 
tation Jf := {i : P(Xt+i = ei\F) > 0}. Writing if = [tpi], we see, from the prop¬ 
erties of given in Definition 12.81 that fji > 0 e*AtXt > 0 i G Jf, 

and hence that V'i = 

= X! -E[X(+i I Ji]). (2.10) 

*eJf 


Write S[ := {(z — z')*(ei — E[W-i-i : * S Jf}, where the conditional 
expectation is evaluated on F. Note that 


E[(z - z'nXt+i - E[Xt+i I Ft]) I = 0, (2.11) 

and hence that min < 0 and max {S[) > 0. If min (S'f') < 0, then (12.1011 
implies that 

/(w, t, y, z) - f{uj, t, y, z') > min {St). 

If min (5'f’) = 0, then (12.1111 implies that Sf = {0}, and it clearly follows from 
(12.1011 that 

f{u},t,y,z) - f{uj,t,y,z') = 0. 


This establishes the conditions of Proposition [221 


□ 


The following lemma is a discrete time version of [2J Lemma 2]. 

Lemma 2.11. Fix ^ G (0,1). Let {/“}ugtY be a family of ^-balanced drivers 
that are independent of y, and for each u &U let '(/'“(w, t, z, z') he the random 
field associated with /“. Let 


g{uj,t,z) := inf {/“(a;,t,z)}. 

u^U 

Suppose \g{uj, t, z)| < oo for all t, z and almost all uj. Then g is also ^-balanced, 
and similarly for sup.|^gjY{/”}- 

Proof. By property (i) of Definition 12.81 the functions {/“}uew are continuous 
in z, uniformly in u. It follows that y is a continuous, and hence measurable 
function of z. 

Fix < > 0. As Ft can be made up from finitely many atoms, there exists 
a single null set Nt G Ft such that N' C Nt for every null set N' G Ft- For 
simplicity, define y(w, t, z) = 0 for u) G Nt. 

For every oj ^ Nt and every z,z' G R^, we can find a sequence {«„}„>! in 
U (dependent on z,z') such that, omitting the arguments u!,t for clarity, 

lim (z-z')>“”(z,z') = inf |(z - z')>“(z, z')}. (2.12) 

For each n, (z, z') is a stochastic vector, i.e. it belongs to the compact subset 
{ip G '■ Pi > It, Sill Pi — !}■ Hence, there exists a subsequence {un,,}k>i 
such that 

lim ■0“”'= (z, z') = ^(z, z'), 

k—¥oo 


11 








where z') is a stochastic vector which satisfies conditions (ii) and (iii) of 
Definition 12.81 In addition, '0 can be constructed in such a way that it is 
measurable in z,z' (see [H Theorem A.10.5]). Further, by (12.121) . we have 


(z - z')*^{z, z') = inf Uz - 2 ')}. 

u^lA 

By a similar argument, there exists a stochastic vector '4}{z, z'), which also sat¬ 
isfies conditions (ii) and (iii) of Definition l2.81 is measurable in z, z', and is such 
that 

(z — z')*'!/j(z, z') = sup {(z — z’)*ll)'^{z, z^)}. 

U^IA 

We have, for each u &U, 

r{z) - /“(zo = (z - zo* {r{z, z') - AXt) , 

from which we deduce that 

(z - z')*'ip{z, z') < g{z) - g{z') + {z - z')*AtXt 
< (z - z')>(z:,z'). 


With 0/0 := 1/2, it follows that 

jz - zTi’jz, z') - jgjz) - g(z')) - (z - z'YAtXt 
^ {z- z')*{tj}{z,z')-ip{z,z')) 

Define 'tp{z, z') := ^^(z, z') -|- (1 — g)'4>{,z, z'). Then 

giz) - 9{z') = {z- z')* (V’(2, z') - AtXt ), 

and we see that the random field z, z') satisfies all the conditions of 

Definition 12.81 for the driver g, and hence that g is 7 -balanced. 

The proof for sup„g^|{/“} is similar. □ 

2.3 Markovian solutions 

We showed in Lemma 11.71 that, if A, / and the terminal condition ^ are all 
Markovian, i.e. at time t they only depend on w through the value of Xt, then 
so is the solution {Y, Z) of the corresponding finite horizon BSDE. We shall now 
extend this result to our infinite horizon equations. As previously, we assume 
throughout that A is a Markov chain with transition matrix At- 

Lemma 2.12. Let a > 0, let f be a Markovian driver satisfying the conditions 
of Theorem \2.5[ and let {Y, Z) be the unique bounded solution of the associated 
discounted BSDE. Then there exists a deterministic function r; : N x A —>■ R 
such that, for all t, Yt = v{t,Xt) and e^Zt = v{t -|- 1,6^) for all Ck € A. 

Moreover, writing Vj+i for the vector in K'^ with entries e^Vt+i = v{t + 
l,efc), the function V satisfies 

(1 + a)v{t, Ck) - f{ek, t, vt+i) - v/+Mtefc = 0 (2.13) 

for all t and all Ck & X such that P(At = Ck) > 0 . 
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Proof. Let (F^, Z'^) denote the (unique adapted) solution of the finite horizon 
BSDE (12.61) . Note that we are in the situation of Lemma \m with (fi = 0. 
Hence, for each T > 0, there exists a function : {0,1,..., T} x X —>■ R such 
that = v'^{t,Xt) and elZf = v'^{t + 1,6^) for all Ck G X. 

Recall, from Theorem 12.51 that Yt as T ^ oo with respect to the L°° 

norm, uniformly on finite sets in t, from which we deduce the existence of a 
function u : N x ^ R such that 1 * = v{t, Xt) for all t. 

Note that the one-step dynamics of the infinite horizon BSDE (12.51) are 
identical to the one-step dynamics of the corresponding finite horizon BSDE 
(EH). Therefore, given the value of Yt+i, we can construct the value of Zt exactly 
as in the finite horizon case. In fact, since Yt+i is Markovian, we can apply the 
same argument as in the proof of Lemma [T771 to deduce that = v{t 1, e^) 
for all Cfe G X. The equation (12.131) follows from the one-step dynamics of 
( 1 ^ . □ 

Corollary 2.13. Recall the conditions of Lemma \2.1‘A Suppose that X is 
time-homogeneous, so that A = At does not depend on t, and that the driver 
f = f{Xt,z) is also independent oft. Then the same is true of the function v 
given in Lemma \2.12l 

Proof. The discounted BSDE (12.51) is given in this case by 

YT=Yt- Y. {fiX^,Zu)-aYu)+ Y KiXu+i - AXu). 

t<u<T t<u<T 

Note that, given the summands are independent of u. By the uniqueness 
of solutions established in Theorem 12.51 the same must therefore be true of the 
solution {Yu, Zu), and the result follows. □ 


3 Uniformly Ergodic Markov Chains 

The main aim of this section is to obtain ergodicity estimates for a discrete time 
Markov chain which hold uniformly for a suitable class of perturbations of its 
transition matrix. We will loosely follow the argument given in Section 3 of [B] , 
by adapting the main steps into the discrete time setting. Henceforth, we will 
assume that all the Markov chains we consider are time-homogeneous. 


3.1 Uniform ergodicity 

The following lemma will turn out to be a useful tool for improving bounds on 
expectations, and improves slightly the estimate given in 0 Lemma 3.3]. 

Lemma 3.1. Let T be a random variable, and consider G{f)) = sup^, [e^^], 
where v is a parametrisation of probability measures under which the expectation 
is taken. Suppose there exist finite constants f3*,K > 0 such that G{j3*) < K. 
Then, for any e > 0, 


G{P) < 1 + e 


for all 


/3e 
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Proof. Let 0 < c < (-^ A l), so that < 1 + ce^*^ for any T > 0. Then, 

for any given measure v in the class considered, we have 

Ejy < El, + ce^ < 1 + cEj, [e^ < 1 + e. 

Taking the supremum over all measures v, we deduce the result. □ 

Definition 3.2. Let M{X) denote the vector space (over R) of finite signed 
measures on {X,V{X)). We will endow this space with the total variation norm, 
given by 

xex 

for /r G M{X). By standard results, M{X) is a Banach space. Let M denote 
the set of probability measures on {X^'P{X)). Note that At is a closed subset 
of At (A). 

Definition 3.3. Let A be a Markov chain on X, and P* its transition operator, 
so that Pt/J, is the law of Xt given Xq ^ /i. We say that the chain X is uniformly 
ergodic if there exists a probability measure tt on X, and constants R, p > 0 
such that 

sup \\PtP — t^Wtv < Pe“^‘ for all t >0. 
ueM 

In this case tt is the unique invariant measure for X. 

Under our assumptions of discrete time chains on a finite state space, we 
have the following simple classification of uniformly ergodic chains. 

Proposition 3.4. Let X and Y be two independent copies of a Markov chain 
on the finite state space X, and let T = inf{f > 1 : A( = Yt\ he the first meeting 
time of the two chains. Then the following statements are eguivalent: 

(i) X is uniformly ergodic. 

(ii) X is irreducible up to transient states (i.e. X has only one closed commu¬ 
nicating class) and is aperiodic. 

(Hi) There exists j5 > 0 such that 

G*{j5) := sup Ea;^[e^^] < oo, 

where ^xy denotes expectation conditional on Xq = x and Yq = y. 

Proof. The implications (i) => (ii) => (Hi) are straightforward. The implication 
(Hi) => (j) will be demonstrated in the course of the proof of Theorem 13.51 □ 

Note that, by Lemma l3.ll we can make G*{(3) arbitrarily close to 1 by 
choosing (3 sufficiently small. 
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3.2 Split chains 

Recall from Definition l2.61 that we say B is 7 -controlled by A, and write B A, 
a B > jA componentwise, and we write A B if both B A and A B. 

Theorem 3.5. Let A and B be transition matrices such that B is j-controlled 
by A for some 7 S (0,1). If, under the measure induced by A, the chain X is 
uniformly ergodic, then X is also uniformly ergodie under the measure induced 
by B. Furthermore, the constants R and p in Definition \3.S\ depend only on A 
and 7 , and R can be made arbitrarily close to 1 (with a corresponding decrease 
in p). 

In our discrete time, finite state setting, the fact that X is uniformly ergodic 
under the measure induced by B is trivial. The interesting part of Theorem 13.51 
is the claim there exist constants of ergodicity R, p which hold for all transition 
matrices B such that B A. We shall make use of this result in Section 4, in 
the proof of existence of bounded Markovian solutions to Ergodic BSDEs. 

The proof of this theorem is the main purpose of this section. To do this we 
will use a Nummelin splitting argument, the broad theory of which is described 
in Meyn and Tweedie [H Chapter 5]. We will assume the conditions of the 
theorem throughout the remainder of the section. 

Definition 3.6. Define the split space of T to be T := X x {0,1}. We will also 
denote the layers of the splitting by Xq := X x {0} and Xi := X x {!}, so that 
X = Xq U Xi. Given y € X, denote the corresponding elements in Xg and Xi 
by yo and iji respectively. 

Let v G M{X). For each y € X, let 

Him}) = (1 - vHyi}) = iv{{y}). 

This defines a split measure v on Similarly, we define the splitting 

of a column vector (f G via the splitting map n : ^ defined by 

n((/)) ;= (1 - 7 ) 

where here 0 is the zero vector in K^. 


(1-7)0 

70 


5 2 AT 


Remark 3.7. Note that these splittings are consistent in the following way. Con¬ 
sider a measure 1 / as a vector in K.-^, so that v{{y}) = y*v for each y € X. 
Similarly, consider the split measure h as a vector in so that 


Him}) 





0 

y 


* 


V 


for each y G T. Then we see that the split measure v corresponds precisely to 
the split vector ri(i 2 ). 

Notice that ifis a probability measure on X, then v \sa probability measure 
on X. Equivalently, if 0 is a stochastic vector in M.^, then 11(0) is a stochastic 
vector in 


Definition 3.8. We can define the splitting of a matrix by splitting its columns 
as in the previous definition. That is, given a matrix M, define 


M := 


(1 -7)M 
7M 
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Definition 3.9. Recall the assumptions of Theorem 13.51 in particular that A 
and B are transition matrices with B A for some 7 € (0,1). Let 


C = (1 - 7)-'(S - 7^), 

and let 6 = [C | A], i.e. the augmented matrix formed from C and A. Note 
that C and B are both transition matrices. 

We can now define a new Markov chain X on the split space X, which 
jumps according to transition matrix B. Intuitively, transitions occur from a 
state X € Xq according to the vector Cx, and from a state x G Xi according 
to the vector Ax (where x is the projection of x in T), except that the result 
is randomly split between the layers Xq and Xi with probabilities 1 — 7 and 7 
respectively. 

Lemma 3.10. Let </> G K.'^. Then n(B(/>) = 6n(0). 

Proof. By the definitions of n, B and C, we have 

Bn{(j)) = {1 - j)C(j) +'yAcf 

= (1 - j)Il{Ccj)) + ^t\{A(j)) = t\{B(j)). 


□ 

Proposition 3.11. Let X be a Markov chain on X with initial distribution 
Xq ~ V and transition matrix B, and let X be a Markov chain on X with initial 
distribution Xq ~ i> and transition matrix B. Then X and X have the same 
marginal distributions, ignoring the splitting. That is, for any t and any y G X, 

y*E[Xt] = F(Xt =y)= F{Xt G {yo, yi})- 

Proof. Consider v as a vector in and v as a vector in as in Re¬ 

mark 13.71 By the Chapman-Kolmogorov equation, the distribution of Xt is 
given by E[Xt] = B^v, and the distribution of Xt is given by E[Xt] = B*'0. 
By applying Lemma [3.101 1 times, we obtain n(i3‘^) = B*Il{i^). Recall from 
Remark 13.71 that = V. Hence, we have n(E[Xt]) = E[W], and the result 
follows. □ 

The following lemma follows easily from the definition of B and the basic 
notion of conditional probability. 

Lemma 3.12. Let X be as in the above proposition. Write E* for the expec¬ 
tation conditioned on X never leaving the layer it starts in (i.e. never jumps 
from Xq to Xi, or from Xi to XqJ. Then, under E*, X jumps according to the 
transition matrix 

' C O' 

0 A ■ 

Intuitively, under E*, X either jumps in Xg following transition matrix C, 
or in Xi following transition matrix A, depending on which layer it starts in. 
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3.3 Exponential moment bounds 

We will now consider two independent copies of our Markov chain on the split 
space, and show that the first meeting time of these chains admits exponential 
moments. 

Theorem 3.13. Let X and Y be two independent Markov chains on the split 
space X, each following transition matrix B, as defined in Definition 1 ,9. .91 Let 
S := inf{t > 0 : = Yt}- Then, for any e > 0, there exists fi > 0 such that 

H*{P) := sup < 1 + e for all fi € [O, ,5], 

where is the expeetation conditional on Xq = x and Yq = y. Furthermore, 
(3 does not depend on B except through A and 7. 

The proof of this theorem will be done in a number of steps. First, we will 
introduce some more notation. 

Let Kt denote the number of jumps, up to time t, that result in both X and 
Y being in Xi when they were not both in Xi previously, and denote by tk the 
time of the such transition. That is, let tp = 0 and 

tk = inf{t > tk-i : iXt,% e X^) \ {Xt-uYt-i e Xi)} 

for fc > 1. Finally, let 

f = mi{t >l:{Xt=Yt€ Xi) n (Xt-i,Yt.i e Ti)}, 

i.e. the first time the chains meet in X\, both having jumped from a state in 
Ti. 

Lemma 3.14. Let G*{(3) he the function defined in Proposition \3.4\ that is, the 
supremum over starting states of the moment generating function of the first 
meeting time on the basic (unsplit) state space X of two independent Markov 
chains with transition matrix A. Then, for any x,y & X, 

00 

Eiy[e^^] = (3.1) 

k=0 

where is the probability conditioned on Xq = x and Yq = y. 

Proof. Note that, conditioned on {Kf = k}, X and Y do not leave Xi between 
tk, the time they arrive in Xi when they were not both in Xi previously, 
and T, the time at which they next meet in Xi. Write E* for the expectation 
conditioned on both X and Y not leaving the layers they start in. Then 

Exg \Kf = k]< sup E:^ = G*{2fi), 

x,y£Xi 

since, by Lemma 13.121 under E*, X and Y jump in Xi following transition 
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matrix A. Then 


<Eip[ 

< ]E4y[ 
= Eiy[ 

< ^xy[ 

and the result follows. 




^E^y (e^ 5 [e/5(^-‘'=) I 4]y 

^E^y [e^ 5 [e2'5(^-*'“)l{K^=fe} I 4] 

^ I Kf = k]¥iy{Kf = k 

e^P^’^Y{G*{2p)¥^y{Kf = k)Y, 


□ 


We shall now seek to bound the components of the sum in Lemma Fd. 141 

Lemma 3.15. For states x,y G X, consider a pair of independent basic (un¬ 
split) chains X,Y following transition matrix A, with starting values Xq = x 
and Yq = y. For n > 1, let q(x, y; n) denote the probability that the first positive 
time the chains meet is at time n. Let 

OO 

97 := inf Y] q{x, y, n)7^", 

n—1 


where we recall that 7 G (0,1). Then G (0,1) and, for any x,y G X and any 
k > 0, 

F,iyiKf = k)<j^l-q^)’^-\ 

Proof. The fact that q-y G (0,1) is clear, as we are working with a finite state 
space, and the independent chains almost surely meet after a finite time. 

Now, suppose first that x,y G Xi. Then, under Pjj, {Kj. = 0} is the event 
that the split chains X and Y meet before either of them leaves Xi. As they 
are independent, at each transition the probability that both chains will jump 
to Xi is 7 ^. Further, by Lemma F3.121 for as long as the chains remain in Xi, 
they jump following transition matrix A. Hence, 

00 

Eiy{Kf = 0) = ^g(a;,j/;n)72” < 7 ^. 

n—1 

Now let fc > 1. Each time the chains both arrive in Xi when they were not both 
in Xi previously, they will either meet before one of them leaves Xi , or they will 
not. Considering these as successes and failures, a geometric trials argument 
yields 


^Xyi^Xrp - /c) - E^;^ 


fc -1 


^ qiXt ,, Yt ,; n) 72 " H U " E 


2=0 


<7^(l-97)^ 


where, in the notation of Definition 13.61 we define q{xi,yi;n) = q{x,y;n) for 
notational simplicity. Now suppose that either 3: G Aq or y G Aq. Clearly the 
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chains must first both arrive in A'l before they can meet in so ¥xy{Kf = 
0 ) = 0. For A: > 1, we have 


— ¥ixy 

v,2 


k-1 


, Yt ,; n)72- M] F " E 5 


, n—1 


n—1 


< 7 (1 - Qj) 


k-1 


Putting this together, we deduce the result. 


□ 


Lemma 3.16. Let q-y be as in the previous lemma. Then there exists /3 > 0 
such that, for any x,y € X and any k > 0, 





k 


Further, this /3 only depends on 7 and . 

Proof. It will be useful to consider two independent geometric random variables 
Qi ^ Geom(l — 7 ^) and Q 2 ^ Geom( 7 ^), 


where we adopt the definition of the probability mass function of Q Geom(p) 
as being given by F{Q = k) = {1 — p)^~^p for A: > 1. 

If i, y G Xi then ti is the time taken for either of the chains to leave Xi, plus 
the time taken for both chains to return to Xi. It follows from an application 
of the strong Markov property that these times are independent. We therefore 
have that G = Qi + Q 2 - If either x € Xq or y £ Xq then G is just the time 
taken for both chains to arrive in Xi, so in this case ti = Q 2 . Similarly, for 
k > 2, tk — tk-i is the time taken for either of the chains to leave Xi, plus the 
time taken for both chains to return to Xi, and these times are independent, so 
it follows that tk — tk-i = Qi + Q 2 - Hence, for all A: > 1, 

<E[e2/3(Qi+Q2)], 

By Lemma |3.11 we can make a new choice of /3 > 0, which depends only on 7 
and qy, such that 



It follows from another application of the strong Markov property that tk — 
tk-i is independent of Ai,..., tk-i for each k. Then, for A: > 1, 





k 


□ 


Proof of Theorem 17.1,91 Substituting the bounds given by Lemmas !,!.Ihl and fS. 161 
into (ED, we obtain 




1 



V1 - 97 y 



19 

















This is valid provided that j3 is chosen to be at least as small as the /3 given in 
Lemma [3.161 The expression above is finite provided that /3 is also sufficiently 
small to ensure that G'*(2/3) is finite, which can be guaranteed by Proposi¬ 
tion [Til Note that this choice of /3 depends only on A, 7 and q^. 

With this /3, we have 

H*{f3)= sup < sup Exy[e^'^] < 00 . 

x^y^X x,y^X 


By Lemma [3.11 for any e > 0 there exists ,5 > 0 such that < 1 -|- e for 

all /3 S [ 0 , /3]. Finally, we see that /3 depends only on A, 7 , e, and on q^, which 
is itself a function of A and 7 . In particular, /3 does not depend on B except 
through A and 7 . □ 

Corollary 3.17. Let X and Y be two independent copies of the Markov chain 
on X with transition matrix B. Let T = inf{t > Q : Xt = Yt\ be the first 
meeting time of these chains. Then, for any e > 0, there exists j3 > 0 such that 

sup Exy [e^^] < 1 -f e for all (3 e [O, P \, 

x,yGX 

where, as usual, E^y is the expectation conditional on Xq = x and Yq = y. 
Furthermore, (3 does not depend on B except through A and 7 . 

Proof. By ProDOsition l3.111 X has the same marginal distribution as X, ignoring 
the splitting, and similarly for Y. Hence T, the first meeting time of X and Y, 
is less than or equal (in distribution) to S, the first meeting time of X and Y, 
which we showed has the required bound in Theorem 13. 131 □ 

Proof of Theorem \3 .51 Let X and Y be two independent copies of the Markov 
chain on X with transition matrix B. Let T = inf{t > 0 : Xt = Yt} he the first 
meeting time of these chains, as in the previous Corollary. Let fj,,v G M (recall 
Definition I3.2|l . Write Pt for the transition operator of X (and Y), so that Ptp, 
is the law of Xt given Xq ^ fi, and Ptiy is the law of Yt given Yq v. It is easy 
to see that, conditioned on {T < t], Xt and Yt have the same distribution. It 
follows that 

\\Ptgi - PM\tv < IP(T > t\Xo p.,Yo ^ v). 

By Corollary 1 3. 171 for any e > 0, there exists /3 > 0, which depends only on A, 
7 and e, such that 

sup Exy [e^^] < 1 -I- e. 

x,y^X 

Then, by Markov’s inequality, 

\\Ptp. - PtvWxv < P(T > t\Xo gL,Yo ^ v) 

< E[e^'^ I Xo - Ai, Ho - < (1 -b e)e-^*. (3.2) 

Let s > t. Replacing u by Pg-tM in the above, we obtain 

||PtM — PsFWtv < (1 + e)e~^*. 
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Since this holds for all s > t, we see that {Pt^J'}t>o is a Cauchy sequence in M, 
and hence converges to an element tt € M, and from (13.21) . tt is independent of 
the initial measure /r. Taking the limit as s —>■ oo in the above, it follows that 

sup \\PtfJ. - ttWtv < (1 + 

iJ.eM 

Hence, a Markov chain under the measure induced by B is uniformly ergodic. 
Further, as noted above, the rate of convergence [3 does not depend on B except 
through A and 7 , and we may take e arbitrarily small, with a corresponding 
decrease in ,5. □ 

4 Ergodic BSDEs 

We now introduce an ‘Ergodic BSDE’ (EBSDE) as an infinite horizon equation 
of the form 


rT = rt- ^ (/(x„,z„)-A)+ ^ (4.1) 

t<u<T t<u<T 

A solution of this equation is a triple (Y, Z, A) such that (14.ip holds a.s. for 
all finite values of t and T such that 0 < t < T, where, as usual Y and Z are 
adapted processes of appropriate dimension, and where A G K is a constant. 
Note that, unlike in our discounted BSDEs where the constant a is given, here 
A is to be found as part of the solution. 

We will follow the method given in Section 4 of [B], which is itself based 
on the work of m and [5]. Throughout this section we will assume that X 
is a uniformly ergodic, time-homogeneous Markov chain on X with transition 
matrix A, and that the driver f : X x K.^ —>■ R is 7 -balanced, Markovian, and 
independent of y and t. Under these assumptions, we shall prove a result on 
the existence and uniqueness of solutions to the EBSDE (14.11) . 

Lemma 4.1. Under the assumptions stated above, let Z,Z' be any two deter¬ 
ministic, or Markovian, -valued processes which do not depend on t, defined 
up to equivalence ^m- Then there exists a probability measure Q on (U, P) such 
that the following conditions hold: 

• Q is equivalent to P on Pt for every finite t, 

• Mt:=- Eo<„<t if (Xu, Z^) - /(X„, Z'J) + j:o<u<tiZu - ts 

a martingale under Q, and 

• under tj, X is a uniformly ergodic Markov chain, and the constants of 
ergodicity R, p depend only on A and 7 . 

Proof. Let t, z, z') be the random field associated with /, as given in Def¬ 
inition 12.81 By our assumptions on / and the processes Z, Z', there exists a 
(deterministic) matrix such that 

i:{uj,t,ZuZ'^ = ^^Z'x^ 

holds for all t. It follows from the properties of ip given in Definition 12.81 that 
is a transition matrix, and that 
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Let Q be the probability measure under which X is a Markov chain with 
transition matrix . Since A, the matrices and A have the 

same pattern of zero entries, so that the possible jumps of X are the same under 
Q and P. As we are in a discrete time, finite state setting, it follows immediately 
that Q is equivalent to P on Xt for every finite t. We also have that 

EQ[{Zt - z'^yut+i I Xt] = {Zt - - AXt) 

= f{Xt,Zt)-f{Xt,zy, 

from which it follows that EQ[Mt+i — Mt \ -At] = 0 , so that M is a martingale 
under Q as required. The final statement is the result of Theorem l3.5l □ 

Remark 4.2. As we saw in Lemma [2.121 and Corollary 12.131 the Z part of the 
solution of our discounted BSDE is constant (i.e. deterministic and independent 
of t), though the result above holds just as well for any Markovian processes 
Z, Z', provided they are still independent of t. If they do depend on t then 
X will in general be a time-inhomogeneous Markov chain under Q, in which 
case the notion of uniform ergodicity is no longer meaningful. Nevertheless, 
in an analogous context, [5] uses the continuous time version of the ergodicity 
estimates we obtained in Section 3 (as given in 0) to prove the existence of 
solutions of continuous time BSDEs up to unbounded stopping times. 

Lemma 4.3. For a > 0, let (E“, Z°‘) be the unique bounded solution (as given 
in Theorem \2.5\} of the discounted BSDE 

Yy=Yy- ^ (/(X„,Z“)-ay„“)+ ^ (4.2) 

t<u<T t<u<T 

and let xq G X be an arbitrary state. By Lemma \2.1‘A and Corollarv \2. DA there 
exists a function : A —>■ M such that = v°‘{Xt) and = u“(efe). Then 
there exists a bound C > 0 such that 

|u“(a;) — u“(a;o)| < C", a|u“(a;)| < C" (4.3) 

uniformly in x and a, and hence there exists a sequence q;„ \ 0 such that 

(u“"(a:) — u“"(a;o)) ^ u(a:) and q;„u“" (x) —^ A for all xGX, 

for some A S R and some function u : A ^ M. 

Proof. Let C > 0 be a bound on !/(•, 0)|. It then follows from Theorem 12.51 that 
|u“(-)| < C/a. By Lemma l4Tl there exists a measure Q“ such that 

- ^ (/(A„,Z“)-/(A„, 0 ))+ 

0 <u<t 0 <u<t 

is a martingale under Q“. Moreover, X is uniformly ergodic under Q“, and the 
constants of ergodicity R,p do not depend on a. Writing PjfSx for the law of 
Xt under Q“ given Xq = x, we see from the proof of Theorem 13.51 that 

\\P^5x-P^5x'\\tv<R^~'’' 

for all x,x' G A. From the one-step dynamics of (14.211 . we obtain 
Eq. [u“(W+i) I Tt] = (1 + a)v^{Xt) - fiXt, 0). 


22 










It follows by induction that 


n“(Xo) = Ec 


T-l 


(1 + ay^v^iXr) + ^ (1 + 0 ) 


k=0 


Xo 


for all T > 1. As |?^“(-)l < C/a, letting T —)• oo, we deduce that 


v°‘{x) = lim Er 

T —^■oo 


T-l 


^(l + a)-('=+i)/(Afe,0) 


fc =0 


Xq=x 


Then, for any x, x' G X, we have 


|x“(x) -?;“(x')| = 

lim Eq^ 

T-vcxd 

'T-l 

^(l + a)-('=+i)/(Afc, 0 ) 

1 - 

II 



-1 


— lim Er 

T-too 


T-l 


Xn = x' 


^(l + a)-('=+i)/(Xfe,0) 

k^O 

+ [ /(Afe,0)(d(P,“4)-d(Pfe“4')) 

T-l 

< lim 21:7 V(l + a)-('=+^)||Pfc“(5,,-Pfc“(5, 

T—i-mo 


= lim 

T^a. 


T-l 


k=0 

T-l 


< lim 2 C'Py^(l + a) 

“ T-^oo ^ 


We therefore have the bound 


-(fc+i)g-pfe = 


\TV 


2CR 


k=0 


1 + a — e P 


‘(x) < 


2CR 

1-e-P’ 


which holds for all x,x' G X and all a > 0. Since a|u“(-)| < C (and we 
may certainly assume that R > 1), we see that (14.31) is satisfied with C = 
2C'P(1 - e-^)-h 

By the Bolzano-Weierstrass Theorem, there exists a sequence a„ \ 0 such 
that (xo) —>■ A and 

(u“"(x) - z;“”(xo)) v{x) 

for all X € ff, for some A € M and some function u : A —>■ K. Note that |A| and 
|n( • )| are both bounded by C. Finally, we notice that, for any x G X, 

anV°‘" (x) = anV°‘" (xo) + a„ (x) - (xq)) -)• A, 

so the convergence of this sequence to A holds for all x. □ 

We are now in a position to prove existence of solutions to our Ergodic 
BSDE. 


Theorem 4.4. Let v and A be as constructed in Lemma \/.S\ The triple (Y, Z, A), 
where 

Yt := v{Xt), elZt := v{ek), 
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is the unique bounded, stationary (i.e. does not depend on t), Markovian solu¬ 
tion, with v(xo) = 0, to the Ergodic BSDE 

YT = Yt- ^ (/(X„,Z„)-A)+ ^ (4.4) 

t<u<T t<u<T 

Any other bounded solution (Y', Z', X') satisfies X = X', and any other bounded, 
stationary, Markovian solution (Y', Z',X') satisfies Yt = Y)' + c for some c S R, 
and Z '^M Z'. 

Proof. Let {a„}n>i be the sequence constructed in Lemma [L3l We have that 
and = u“"(efc) solve the discounted BSDE 

t<u<T t<u<T 

However, since ||l||Mt+i = 0, and / does not distinguish between values of Z^ 
up to equivalence we can equally write — u“"(a;o)l in the place 

of in the above. Note that — u“"(a;o) —>■ as n —>■ oo for each 

Cfc £ X, and that, by the bound established in Lemma 021 |e^Z“" — 
is uniformly bounded. Since / is Lipschitz in z, we deduce that 

- v“"(a:o)l)as n ^ oo a.s. 

It follows that 


Yt= lim (u“"(XT)-u“"(a;o)) 

n—^oo 

= hm (t,“"(Xt)-u“"(a:o)) - lim V (/(X„, Z“") - (X„)) 

n—¥oo n^oo ^ 

t<u<.T 

+ hm ^ {zf:-yM,,+^ 

n—xx) ^ ' 

t<u<T 

= v{Xt) - lim V (/(X„, (a;o)l) - (X„)) 

n—^oo 

t<u<.T 

+ lim yy (Z“" - v“"(a;o)l)*M„+i 
n^oo 

t<u<T 

= Yt- Y. {fiXu,Z,,)-X)+ Y KMu+u 


t<.u<.T 


KuKT 


and we see that (Y, Z, X) is indeed a solution of the EBSDE ()4.4E 

Suppose that (Y', Z' ,X') is another bounded solution. Let Y = Y — Y', 
Z = Z-Z' andX = X-X'. Then 


Yt = %- Y (/(^«,^J-/(^«,0-A)+ ^ (4.5) 

0<u<T 0<u<T 

By Proposition 021 there exists a measure Q such that 

- ^ (/(x„,z„)-/(x„,z;))+ ^ 

0<u<t 0<u<t 
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is a martingale under Q. Taking an Eq expectation in (14.5p . we obtain 

X = T-^Eq[Yt-Yo]. 

Since Y is uniformly bounded, taking the limit T —>■ oo gives A = 0, so that 
A = A'. Substituting back into (I4.5p and taking an Eq[ ■ | Fq\ expectation gives 

EQ[fT|^o] =hb. (4.6) 

Suppose further that Y' and Z' are Markovian and do not depend on t, so that 
in particular there exists a function : A” —>• K such that Y/ = v'{Xt). Then 
the measure Q may be taken to be the measure given by Lemma 14.11 so that 
X is still a uniformly ergodic Markov chain under Q. Writing if for the ergodic 
measure of X under Q, it follows from (14.61) that, for any x & X, 

v{x)-v'{x)= lim Eq [Ft I ^0 = a:] = [ {v{y) - v'{y))di{y). 

T-J-OO 

Since the right hand side is independent of x, we see that v{x) = v'(x) + c for 
all X, and hence that Yt = F/ + c, for some c G R. In particular, if x'(xo) = 0, 
then c = 0, and hence F = F' up to indistinguishability. 

With A = 0 and Yt = c = Ft+i, we deduce from the one-step dynamics of F 
that Z^Mt+i = 0, and hence that Z Z'. □ 

Corollary 4.5. The sequences {anV°'''(x)}n>i and (x) —(xo)}rt>i con¬ 
structed in Lemma \4-d\ converge to A and v(x) respectively for any choice of 
sequence \ 0. 

Proof. Suppose this were not the case. Then we could construct two sequences 
with distinct limits, both of which would yield bounded, stationary, Marko¬ 
vian solutions to the EBSDE, contradicting the uniqueness established in The¬ 
orem 14.41 □ 

Corollary 4.6. Let n denote the ergodic measure of X under ¥. Writing v for 
the vector in K'^ with entries e^v = v{ek), the value A in the EBSDE solution 
(F, F, A) = {v{Xt),'v, X) is given by 

>'= [ f{x,v)dn{x) = /(x,v)7r({x}). 

xex 

Eurthermore, there exists a probability measure tTv on X such that 


^ = [ fix,0)dTT^{x). 

Jx 

Proof. The invariance of the ergodic measure tt implies that for any fixed time 
t > 0 and any function g : K, 


[ E[g{Xt) |Xo = x]d 7 r(x) = [ g{x)d'K{x). (4.7) 

Jx Jx 


For any T > 0, we have 


,{Xo)=v{Xt)+ Y (/(^«,v)-A)- ^ (4.8) 

0<u<T 0<u<T 
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Then, by the invariance property (14.71) . 


/ v{x)d7T{x) = 

[ E 

v{Xt)+ (/(^«w)-A) 

Xo = x 

Jx 

Jx 

0<ii<T 



dn{x) 


= / v{x)dTr{x)+T / f{x,v)d'!r{x) — TX, 

Jx Jx 


and rearranging gives the first result. 

By Lemma [4.11 there exists a probability measure Q such that, under Q, 


- ^ (/(X„,v)-/(X„,0))+ ^ v*M„+i 

0<ii<i 0<u<t 


is a martingale and X is still a uniformly ergodic Markov chain. Let tTv denote 
the ergodic measure of X under Q. From (|4.8I) . and the invariance property 
63), we have 


v(x)d7rv(x) = 


f X 


f X 


;(Xt)+ ^ (/(X„,0)-A) 


0<u<T 


Xo = X 


dir^ix) 


= / v{x)dTTv{x) + T / f{x,0)dTrv{x) — TX, 
Jx Jx 


and rearranging gives the second result. 


□ 


Remark 4.7. Under the additional assumption that X has no transient states, it 
follows from the one-step dynamics of (14.41) . that the EBSDE solution {v{Xt), v. A) 
is also a solution of the vector equation 


V — f(v) -I- A1 — A*x = 0, (4.9) 

where f(v) denotes the vector with entries e^f(v) = /(efe,v). Further, this 
solution is unique up to equality in A and a constant shift in v. 

We now provide a comparison theorem for Ergodic BSDEs, concerning the 
A part of the solution. 

Theorem 4.8. Let f and f be two ^-balanced Markovian drivers, and let 
(Y, Z, A) and (Y',Z',A') be any corresponding bounded EBSDE solutions. If 
f{x, z) > f'{x, z) for all X € X and z € then X > A'. 

Proof. Write (Y,Z,X) = {v{Xt),x, X) and {Y',Z',X') = (n'(W), v'. A') for the 
corresponding bounded, stationary, Markovian solutions given in Theorem 14.41 
By Lemma [4.11 there exists a probability measure Q such that, under Q, 

- ^ (/'(X„,v)-/'(X„,v'))+ ^ (v-v')*M„+i 

0<u<i 

is a martingale and X is still a uniformly ergodic Markov chain. Write if for 
the ergodic measure of X under Q. Taking the difference of the EBSDEs, and 
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applying the invariance property dUZl), we have 


{v{x) — v' {x))dn{x) 

= / Eq [f (Xt) — v'{Xt) I Xq = x] dTr{x) 

J X 


Er 


lx 


^ (/(X„,v)-/'(X„,v)-(A-A')) 


0<u<T 


Xo = X 


d%{x) 


= [ {v{x) - v'{x))dfr{x)+T [ {f{x,v)-f'{x,v))dTr{x)-T{X-X'), 
Jx Jx 

and hence, by rearrangement, 

X-X'=[ {f{x,v)-f{x,v))d7r{x)>0. 

Jx 


□ 


5 Optimal Ergodic Control 


As an application of the theory we have just developed for discrete time Ergodic 
BSDEs, we shall now present a treatment of an ergodic control problem. We 
will follow the method given in Section 5.2 of [B], which is itself based on the 
work of [TU] and [5]. 

Suppose that 


• Under P, A is a uniformly ergodic Markov chain on X with transition 
matrix A, 

• U is a. space of ‘controls’, which we assume to be a topological space, equal 
to a countable union of compact metrizable subsets of itself, 

• B(') is a continuous function which maps each element u G Id to a transi¬ 
tion matrix such that 5“ some fixed 7 G ( 0 , 1 ), 

• L: XxU^R is a cost function, which we assume to be bounded, 
measurable in x and continuous in u. 


Define the ergodic cost 


J{x,U) = limsup -E^ 

T —>-oo 


E L{Xs,Us) 

.0<s<T 


where 

• U is a W-valued adapted process, which we shall also refer to as a ‘control’, 

• E^ is the expectation under which Xq = x, and for any given pair 
X jumps according to transition matrix 
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We wish to minimize J{x,U) over all controls U. Our approach is to work 
with the probability measure under which X jumps at time t according to 
the transition matrix . As we have made no Markov or time-homogeneity 
assumptions on U, X will in general not be a Markov chain under our new 
measure. 

We define the Hamiltonian 

f{x,z)= inf {L{x,u) + z*{B'^ — A)x}. (5.1) 

Note that / is finite valued and that f{-,0) is bounded. By Lemma 12.111 
f{Xt,z) is a 7 -balanced Markovian driver. Hence, by Theorem 14.41 the EB- 
SDE with driver f(Xt,z) admits a bounded, stationary, Markovian solution 
{Y, Z, A) = (z;(A(), V, A), where as usual v is the vector in with components 
v{ek). 

If the infimum in (EB is attained then, by Filippov’s implicit function the¬ 
orem (see either McShane and Warfield [TT] or Benes [I]), there exists a mea¬ 
surable function k : X x —>■ U such that 

f{x, z) = L{x, k{x, z)) + z*{B'^^^’^'> — A)x. (5.2) 

If the infimum in (15.11) is not attained, then applying Theorem 21.3.4 from 
[U to the function G{x,z,u) = L{x,u) +z*{B‘^ — A)x, we deduce, for any e > 0, 
the existence of a measurable function : X x —>■ U such that 

L{x, Y{x, z)) + z*{B'^ — A)x < /(x, z) + e. (5.3) 

Theorem 5.1. In the setting described above, let (Y,Z,X) be any (possibly 
non-Markovian) bounded solution to the EBSDE with driver f. Then the 
following hold: 

• For any control U, we have J{x,U) > \ = A, with equality if 

f{X„Z,) = L{X„U,) X Z:{B^‘ - A)X, for all s > 0. (5.4) 


• If the infimum in (Ely is attained, then the control Ut = K{Xt, Zt) satisfies 
Jix,U) = A. 

• Even if the infimum in (Ely is not attained, there still exists a control U 
such that J(x, U) = A. 

Proof. The fact that A = A follows immediately from Theorem 14.41 
Note that 

- ^ z;{B^‘-A)x,+ 

0<s<t 0<s<t 

is a martingale under E^. Since {Y, Z, A) is a solution of the EBSDE, we have 
Yt=Yo- {f{Xs,Z,)-X)+ Z*sMs+i 

0<s<T 0<s<T 


for any T > 0. Then 
1 


A = 


Yt-Yo+ L{X„Us) 


0<s<T 


1 


+ ^ E^ E - L(A„ H.) - Z:{B^‘ - A)X,) 

Lo<s<T 


(5.5) 
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Since / is the infimum over all controls, we have 




Yt 


Yo+ Y. HXs,Us) 

0<s<T 


As Yt — ho is uniformly bounded, it follows that 


A < limsup 

T —>^00 


E HXs,Us) 

.0<s<T 


J{x,U). 


If (|5.4I1 holds, then equality holds throughout the above, and we see that 
A = J(x,C/). 

If the infimum in m is attained, then we know that (IE 2 I) holds for some 
measurable function k : X x U. Setting Ug = k(As, Zg), we have that 

(EH) holds, so that X = J{x,U). 

Even if the infimum in (EH) is not attained, by (EH, there exists a measur¬ 
able function t : N x A x — >■ U such that 


L{x, t(s, X, z)) + _ A)x < fix, z) + 2-". 

Setting Us = t(s, Xg, Zg), (15.51) reduces to 


A > 


VT-ro+ Y LiXg,Ug) 

0<s<T 


- y 2-« 

j' 

0<s<T 


and it follows that A = J(a:, 17). □ 

In particular, the controls Ug = K{Xg,'v) (assuming the function k exists) 
and Ug = t(s,Xs,v) are optimal feedback controls. 

Remark 5.2. We note that Ug is time-dependent, but that for any e > 0, a time- 
homogenous feedback control satisfying J{x,tj) < A -I- e can also be attained by 
setting Ug = U{Xg,x), where U satisfies (15.31) . 

It is worth noticing that the A we have obtained is optimal in the class of 
all strategies, not only among those of feedback type. If we had approached 
our problem purely through the equations obtained from (14.91) and (15.21) . this 
optimality would require separate analysis. 

As noted above, (EH) is a sufficient condition to guarantee that J{x, U) = A. 
In [To] and [6], among others, in analogous contexts, it is stated that this is 
also a necessary condition. However, this is incorrect, as modifying the value 
of an optimal control over any hnite time horizon does not affect the associated 
ergodic cost. Indeed, in our setting we have shown that we can always construct 
an optimal control such that (El) does not necessarily hold at any time s. 

Remark 5.3. We have assumed that the Markov chain X and the cost function 
L are time-homogeneous. However, by considering the cyclic classes of X, it is 
not hard to extend the result of Theorem ED to the case when both X and L 
are periodic in time. 
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6 Conclusion 


We have shown that, when our underlying process is a uniformly ergodic Markov 
chain, Ergodic BSDEs with 7 -balanced, Markovian drivers admit bounded Marko¬ 
vian solutions. We have also shown how these equations arise in the context of 
control problems when considering an ergodic cost functional, so that it is the 
long term asymptotic behaviour of the process that is important. 

This has involved extending the existing theory of discrete time BSDEs to an 
infinite time horizon setting, in order to prove the existence of unique bounded 
solutions to a class of discounted BSDEs. We then showed how, by taking a 
suitable limit in these equations, we can obtain solutions to corresponding Er¬ 
godic BSDEs. A notable step in the proof involved finding ergodicity estimates 
for a discrete time Markov chain which hold uniformly for a suitable class of 
transition matrices, a result which is interesting in its own right. 

When considering Ergodic BSDEs, we made Markovian and time-homogeneity 
assumptions on the underlying process X and on our driver function /. However, 
we note that a significant amount of the preceding theory, including our con¬ 
struction of solutions to discounted BSDEs, holds much more generally, without 
any Markovian assumptions. 

Throughout, we have considered discrete time (E)BSDEs as entities in their 
own right, rather than as approximations to the continuous time case. The 
precise connections and the extent to which our results approximate the corre¬ 
sponding continuous time theory remain to be explored. 

One extension of our theory would be to the case where the underlying 
process is defined on a countably infinite state space, though it is expected that 
the majority of our analysis will remain essentially the same. 
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